Skip to content

7.6. Governance

What does governance mean for this agent?

Governance is the ability to state who could do what, under which policy, using which model/prompt/data, which runtime identity confirmed a state change, what happened, and how long the evidence is retained. It is implemented through identities, code, transactions, storage, traces, and review, not a policy paragraph alone.

What does an audit row contain?

CREATE TABLE audit_log (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    ts              TEXT NOT NULL,                         -- ISO-8601 UTC of the action
    actor           TEXT NOT NULL,                         -- executing agent, e.g. "ops-copilot"
    approved_by     TEXT NOT NULL,                         -- ADK user id; synthetic on unauthenticated A2A
    rationale       TEXT NOT NULL,                         -- the approver's stated reason (HITL, Ch. 4.5)
    context_summary TEXT NOT NULL,                         -- current decision context reconstructed at execution
    session_id      TEXT NOT NULL,
    invocation_id   TEXT NOT NULL,
    action          TEXT NOT NULL,                         -- e.g. "restart_service", "resolve_incident"
    target          TEXT NOT NULL,                         -- service name or incident id
    detail          TEXT NOT NULL
);

The action requires ToolContext.user_id, session id, invocation id, and a non-empty confirmation rationale. It trims the rationale, rejects more than 500 characters before writing, then locally redacts concrete PII and obvious credential/token patterns before audit persistence; incident and service identifiers remain intact. A rejected rationale produces neither a mutation nor an audit row.

restart_service and resolve_incident acquire BEGIN IMMEDIATE, read the target and current incident context under that write lock, then mutate and append the audit row in the same transaction. Callers cannot provide context_summary, and an audit-insert failure rolls back the state change.

On the default unauthenticated A2A path, approved_by is ADK's synthetic A2A_USER_<context-id> value. It links the pause, resume, mutation, session, and trace, but it does not identify a real person. The local gateway JWT demonstration validates a caller at the edge without propagating that subject into ToolContext; production governance needs that verified identity handoff.

Is the audit log immutable?

No. SQLite triggers reject UPDATE and DELETE, making rows append-only through the application schema. A storage/database administrator can still replace the file or alter/drop the schema. Production evidence needs restricted administration, external append-only/WORM storage where required, backups, integrity verification, retention, and independent access logs.

How do you inspect action evidence in Kubernetes?

kubectl -n agentops exec deploy/agentops-agent -- \
  python -c 'import sqlite3; db=sqlite3.connect("/app/state/incidents.db"); print(db.execute("SELECT ts, actor, approved_by, action, target FROM audit_log ORDER BY id DESC LIMIT 10").fetchall())'

Compare the returned session/invocation identifiers with the corresponding MLflow trace. Treat the synthetic A2A user as correlation evidence, not authenticated identity. Do not expose audit rows through an unrestricted agent tool.

Which state survives replacement?

The agent and MCP server share the 1 Gi agentops-agent-state RWO PVC, so sessions, A2A tasks, actions, and later reads remain coherent across pod replacement. The agent is the state owner; MCP receives the same claim as a read-only mount, and the backup CronJob reads the source without write authority. MLflow metadata/artifacts use a 5 Gi PVC locally. On GKE, MLflow artifacts move to GCS, while its SQLite backend and agent state remain zonal PVC data.

Skaffold deletion removes course PVCs. A Spot eviction can move/recreate a pod while the zonal PV remains, but zone/cluster/storage deletion and corruption still need a backup/recovery plan that this lab does not ship.

How is cloud authority separated?

  • Workloads that do not call the Kubernetes API—including the agent, MCP server, agentgateway, MLflow, and backup job—do not automount Kubernetes API tokens. GKE Workload Identity Federation uses the metadata server instead.
  • agentgateway WIF identity can consume Vertex/service usage.
  • MLflow WIF identity can write objects only in its artifact bucket.
  • Node identity reads Artifact Registry and performs GKE node duties.

The principle is one workload identity per external authority, with no long-lived JSON key.

What evidence belongs in a release decision?

  • Git/image digest and rendered overlay.
  • dependency/security scan results.
  • offline tests/coverage/adversarial regression.
  • model/prompt/data/scorer lineage and eval results.
  • representative traces and dashboard health.
  • accepted residual risks and approver.
  • rollback/teardown and retention/backup state.

What is the governance checkpoint?

Approve one mock action, verify the action/result/audit row/trace agree, attempt an audit-row update and confirm the trigger rejects it, then document who can still alter the SQLite/PVC. A control is governance evidence only when its bypass/owner and retention are understood.